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Introduction 

This  final  report  details  the  results  of  the  partnership  between  the  Aerospace  Systems  Design  Laboratory 
(ASDL)  at  the  Georgia  Institute  of  Technology  (“Georgia  Tech”)  and  the  Office  of  Naval  Research  (ONR) 
in  helping  to  ensure  the  continued  training  of  the  next  generation  of  engineers  that  will  contribute  to  the 
nation's  ongoing  naval  challenges.  The  Aerospace  Systems  Design  Laboratory  (ASDL)  at  the  Georgia 
Institute  of  Technology  has  been  at  the  forefront  of  the  development  of  advanced  design  methods  and  their 
application  to  revolutionary  concepts.  The  methods  pioneered  include  but  are  not  limited  to  surrogate 
modeling  in  design  applications,  optimization,  robust  design  simulation,  strategic  portfolio  planning, 
technology  impact  forecasting  and  evaluation,  as  well  as  inverse  design  methods  and  capability-based 
design  concepts.  Prior  funding  has  focused  in  developing  the  methods  and  actual  applications  have  been 
incomplete  and  applied  to  simplified  demonstration  problems;  i.e.  portions  of  the  methods  and  techniques 
were  applied  to  problems  that  did  not  reflect  the  full  complexity  of  the  problems  of  interest.  When  the 
methods  were  applied  to  realistic  problems  involving  subject  matter  experts  (SME),  vested  in  a  real  world 
decision  problem,  the  researchers  realized  that  the  processes,  methods  and  techniques  could  be  improved 
significantly.  The  difficulty  lies  in  that  these  prior  applications  are  often  done  under  strict  non-disclosure 
agreements,  therefore  the  findings  cannot  be  published  and  shared.  This  highlights  the  need  to  apply  ASDL 
methods  to  real  world  problems. 

The  researchers  have  identified  various  problems  that  span  the  full  spectrum  of  processes,  methods  and 
techniques  developed  at  ASDL  and  are  significantly  different  to  ensure  the  broad  applicability  of  the 
methods.  The  first  task  is  based  on  an  ONR  N1COP  with  Orizzonte  Sistemi  Navali  (OSN)  tasked  with 
developing  a  process  and  tools  for  integrating  the  evaluation  of  a  ship’s  mission  effectiveness  in  the  design 
process.  OSN  and  its  partners,  CETENA  and  the  University  of  Genoa  collaborated  with  the  Italian  Navy  to 
develop  and  test  an  environment  to  design  ships  and  evaluate  them  according  to  their  mission  effectiveness. 
ASDL  will  provide  guidance  and  apply  the  advanced  design  methods  in  order  to  expand  the  applicability, 
increase  its  computational  efficiency  and  improve  its  ability  to  provide  decision  support  to  the  critical 
stakeholders.  The  second  task  supported  an  analytical  wargame  for  the  US  Joint  Forces  Command 
(USJFCOM)  J9  Division,  tasked  with  evaluating  the  effectiveness  of  the  BMD  system.  This  has  been  listed 
as  one  ofNORTHCOM’s  top  priority  Warfighter  Challenges  for  2011,  J9  and  NORTHCOM  are  investing 
significant  resources  in  developing  the  analysis  tools  to  support  this  effort,  ASDL  supported  the  wargame 
by  developing  real-time  tradeoff  environments  based  on  those  analysis  tools,  provide  technical  assistance 
during  the  workshops  and  host  the  event  in  its  state-of-the-art  collaborative  design  and  visualization 
facilities. 


3 


The  next  project  involved  working  with  Joint  Staff  (JS)  to  in  the  area  of  Command  &  Control  (C2).  C2  is 
a  necessary  and  critical  component  of  war  fighting,  especially  in  today’s  joint  war  fighting  environment. 
C2  is  used  in  the  planning,  directing,  coordination  and  control  of  forces  and  operations  in  the 
accomplishment  of  the  mission,  C2  is  also  largely  responsible  for  enabling  interoperability  in  military 
networks.  Poor  or  non-existent  interoperability  can  lead  to  degradation  in  combat  effectiveness.  The 
objective  of  this  research  is  to  develop  an  analytical  framework  for  understanding  and  exploring  C2 
capability-based  solutions  across  multiple  missions  and  for  multiple  stakeholders,  A  Visual  Command  & 
Control  Capability  Analysis  and  Tradeoff  Suite  (VC3ATS)  was  developed  to  perform  dynamic  Command 
Sl  Control  (C2)  and  mission  analysis  with  Joint  Mission  Thread  (JMT)  architectures  for  the  purpose  of 
comparing  C2  alternatives  in  the  single  and  multi-mission  trade  space. 

The  last  project  involved  demonstrating  and  transferring  Advanced  Design  Methodologies  to  a  challenging 
mission  architecture  problem.  This  involved  also  collaborating  with  the  NASA/Marshall  Space  Flight 
Center  (MSFC)  Advanced  Concepts  Office  (ACO)  in  order  to  enhance  NASA  design  practices  and  enable 
expanded  trade  study  capabilities.  This  effort  focuses  on  applications  for  in-space  disciplinary  analyses, 
Earth-to-orbit  (ETO)  modeling,  and  physics  based  engine  modeling,  enabling  an  encompassing  look  at  a 
conceptual  design  for  Mars  exploration  architecture.  Opportunities  associated  with  advanced 
manufacturing,  materials,  in-situ  resource  utilization  (ISRU),  and  concepts  of  operation  (CONOPS)  w'ere 
evaluated  utilizing  advanced  technology  impact  forecasting  methods.  The  challenge  of  developing  a  fully 
reusable  transportation  architecture  for  manned  Mars  missions  was  selected  in  order  to  provide  a 
demonstration  problem  of  current  relevance  to  NASA,  The  Navy  primarily  benefits  from  being  able  to  take 
the  lessons  learned  from  the  methods  development. 


Objective 

These  tasks  address  significant  real  world  problems  and  were  aligned  with  major  efforts  that  were 
underway.  This  provided  a  unique  opportunity  to  truly  test  methods  developed  in  an  academic  environment. 
From  these  tests  the  methods  were  made  not  only  more  credible,  but  also  easier  to  implement,  more 
streamlined  and  robust. 
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Progress  Statement  Summary 

Funding  provided  by  this  grant  supported  the  full-time  research  activities  of  undergraduate  and  graduate 
students  who  then  became  involved  with  and  contributed  to  ongoing  research  projects  in  complex  System 
of  Systems  (SoS)  design,  engineering,  and  architecting  at  A  SDL.  These  students  engaged  in  relevant 
research  that  was  crucial  in  continuing  the  development  of  Advanced  Design  Methodologies  and  their 
application  to  revolutionary  concepts.  At  the  same  time,  these  students  were  educated  in  state  of  the  art 
systems  design  and  decision  making  methods  such  as  design  of  computer  experiments,  multidisciplinary 
analysis,  advanced  modeling  and  simulation  techniques,  and  multi-criteria/multi-attribute  decision  making. 
The  combination  of  the  naval  experience  and  systems  design  education  will  equip  these  students  with  the 
skills  necessary  to  foster  the  cultural  transformation  that  is  necessary  to  keep  up  with  the  Navy’s  evolving 
challenges. 

Detailed  Progress  Summary 

The  grant  was  organized  into  the  principal  tasks  (1)  PRONTO  ASNET  NICOP  (2)  Advanced  Design 
Methods  in  Support  of  BMD  Wargaming,  (3)  Space  Mission  Architecting  (4)  Visual  Command  and  Control 
Capability  Analysis  and  Tradeoff  Suite  (VC3ATS). 

PRONTO  ASNET  NICOP 

ONR  has  established  a  Naval  International  Cooperative  Opportunities  in  Science  and  Technology  Program 
(NICOP)  with  Orizzonte  Sistemi  Naval i  (OSN)  and  CETENA  (both  Fincantieri  companies)  and  the 
University  of  Genoa.  This  NICOP  is  tasked  with  further  developing  and  testing  the  Application  System  for 
Naval  ship  design  and  Evaluation  and  Testing  (ASNET).  To  provide  a  realistic  test  problem,  ONR  and  OSN 
decided  to  design  an  Offshore  Patrol  Vessel  (OPV).  The  underlying  goal  of  the  project  was  to  allow  the 
designers  of  the  OPV  to  obtain  credible  estimates  of  its  effectiveness  in  conducting  a  wide  range  of 
missions,  e.g,s  anti-air  warfare  (AAW)  to  fishery  patrols,  maritime  interdiction  operations  (MIQ),  as  a 
function  of  the  ship  design  characteristics.  In  order  for  this  to  be  effective,  the  stakeholder  must  obtain  a 
near-immediate  response  from  the  analyses,  which  tend  to  become  more  computationally  demanding  as  the 
analyses  become  more  complex.  This  offered  ASDL  an  ideal  opportunity  to  apply  its  methods  and 
techniques. 

In  order  to  meet  the  technical  objective,  a  rapid  cost -capability  assessment  framework  for  surface 
combatants  was  developed.  This  new  modeling  approach  addresses  multiple  sources  of  uncertainty  during 
conceptual  design  That  were  not  previously  captured  such  as  price  escalation,  fluctuations  in  foreign 

exchange  rates,  and  uncertainty  in  ship  build  scheduling.  The  approach  also  focuses  on  improving  the 
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mathematical  modeling  of  organic  air  asset  availability  for  parametric  ship  designs  under  varying  operating 
conditions.  It  is  important  to  know  when  and  under  what  conditions  air  assets  such  as  rotorcraft  may/may 
not  be  available  to  assist  the  ship  during  operations.  The  approach  also  investigates  improved 
methodologies  for  formalizing  the  implementation  and  improving  the  efficiency  of  Agent  Based  Modeling. 
Specifically,  the  research  seeks  to  address  more  rigorous  ways  of  developing  conceptual  models  for 
operational  scenarios  that  will  lead  to  improved  modeling  and  simulation  verification  and  validation. 

A  graphical  tradeoff  environment  for  rapid,  defensible  and  traceable  assessments  was  created.  This  is  shown 
in  Figure  1. 
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Figure  1.  Graphical  Tradeoff  Environment 

This  tradeoff  environment  captures  both  quantitative  modeling  &  simulation  based  analysis  along  with 
expert-based  qualitative  analysis  to  enable  rapid  relative  cost  estimation  and  risk  assessment.  A  capability 
analysis  is  accomplished  using  an  expert  driven  decomposition  and  criticality  assessment.  This  is  depicted 
in  Figure  2. 
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Figure  2.  Expert  Driven  Decomposition  &  Criticality  Assessment 

A  one  week-long  workshop  was  conducted  with  16  Naval  Officers  ranging  in  rank  from  Rear  Admiral  to 
Lieutenant,  representing  extensive  operational  experience.  The  workshop  provided  a  structured  process  to 
elicit  subject  matter  expert  (SME)  knowledge  to  develop  a  model  of  the  value  of  different  ship  capabilities 
under  a  variety  of  conditions*  This  can  be  seen  in  Figure  3* 
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Figure  3.  Expert-Based  Capability  Analysis 

Overall,  the  integrated  analysis  provides  decision  makers  the  ability  to  trace  the  impact  of  requirements 
from  the  systems  level  all  the  way  up  to  tactical  and  even  high-level  strategic  goals*  This  represents  a 
complete  capability  assessment  that  includes  the  entire  range  of  stakeholder  requirements* 


Continued  development  of  the  assessment  framework  focused  on  three  distinct  areas.  First,  a  new  cost 
modeling  approach  was  fully  integrated  into  the  graphical  tradeoff  environment  to  enhance  both 
probabilistic  cost  estimating  and  budget  risk  mitigation.  Second,  methods  to  improve  modeling  of  the  ship 
motion  was  investigated  in  collaboration  with  Virginia  Tech  in  order  to  help  explore  the  feasibility  and 
validity  of  mathematical  models  for  the  operational  availability  of  aerial  vehicles  operating  from  naval 
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assets.  This  would  result  in  more  accurate  probabilistic  evaluation  models  to  capture  the  effectiveness 
contribution  from  naval  assets  under  varying  operational  conditions.  Lastly,  continued  progress  was  made 
in  the  area  of  operational  modeling  and  integrating  the  analysis  results  along  with  methods  for  additional 
statistical  analysis  into  the  graphical  tradeoff  environment. 

Focus  Area  I:  Probabilistic  cost  estimating  and  budget  risk  mitigation 

At  the  beginning  of  this  phase  of  the  effort  the  cost  estimation  portion  of  the  graphical  tradeoff  environment 
provided  a  confidence  estimation  of  the  total  cost  of  the  OPV  design  by  displaying  cost  using  a  Cumulative 
Distribution  Function  (CDF).  This  is  shown  in  Figure  4. 
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Figure  4.  Original  Cost  Estimation  Display 

Thus,  decision  makers  could  specify  a  desired  confidence  level  when  assessing  the  anticipated  future  cost 
OPV  designs  during  the  decision  making  process.  However,  this  approach  was  limited  to  normal 
distributions  to  model  cost  uncertainty  that  were  specific  to  each  type  of  equipment  captured  in  the  overall 
systems  cost.  In  order  to  grant  decision  makers  the  ability  to  provide  rapid  cost  vs.  capability  tradeoffs 
while  also  dealing  with  multiple  sources  of  uncertainty  present  during  conceptual  design,  a  more 
comprehensive  probabilistic  approach  to  capturing  costs  is  necessary.  These  sources  of  cost  uncertainty  can 
be  categorized  as  follows: 
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■  Weight  estimates  of  Ship  Work  Breakdown  Structure  (S  WBS)  subgroups 

•  Group  100 -Group  700 

■  Regressions  for  Cost  Estimating  Relations  (CERs) 

■  2  sets  of  CERs  -  Product  Oriented  Design  and  Construction  (PODAC)  cost 
model 

■  Future  price  of  commodities  such  as  steel 

■  Cost  of  ship  systems 

•  For  international  navies  especially,  many  mission  systems  are  purchased  from 
foreign  suppliers 

■  Inflation/Escalation 

•  Escalation  defined  as  the  general  increase  in  costs/  prices  over  a  period  of  time 

■  Distinguished  from  inflation  and  the  risk-free  discount  rate 

»  Both  economy-driven  &  customer-driven  escalation  components  can  outpace 
inflation 

■  Requirements  &  regulations 

■  Foreign  exchange  rates 

■  Ship  build  schedule 

By  giving  decision  makers  the  ability  to  distinctly  assess  the  impact  of  different  sources  of  uncertainty 
mitigation  plans  can  be  more  effectively  developed  in  order  to  manage  risk.  This  was  realized  by  creating 
an  updated  cost  analysis  methodology  and  integrating  it  into  the  graphical  tradeoff  environment.  The 
updated  cost  analysis  requires  the  following  inputs 

•  User  Inputs:  Labor  rates,  ship  design  complexity  factor  &  labor/material/equipment  cost 
uncertainty  percentages 

•  Ship  weight  estimations  from  the  ship  design  tab  of  the  graphical  tradeoff  environment 

•  Cost  estimates  grouped  by  systems  from  the  systems  tab  of  the  graphical  tradeoff  environment 

•  Inputs  based  on  regression  of  historical  trends  for  use  in  the  CER  equations. 
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Based  on  these  inputs,  a  more  comprehensive  probabilistic  cost  analysis  can  then  be  performed.  The  results 
of  the  analysis  are  integrated  into  the  graphical  tradeoff  environment.  This  can  be  seen  in  Figure  3, 


IBlIlKlf 


Figure  5.  Comprehensive  Cost  Analysis  Dashboard  Display 

The  end  result  is  the  ability  to  retain  the  rapid  relative  cost  estimation  and  risk  assessment  capability  while 
also  providing  a  comprehensive  assessment  of  risks  associated  with  uncertainties  in  the  CER  regressions, 
parameter  estimates,  foreign  currency  exchange  rate  fluctuations,  and  shipbuilding  learning  curves.  This 
provides  an  even  greater  measure  of  confidence  in  the  cost  estimations  and  confidence  levels  displayed  by 
the  CDF, 
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Focus  Area  2:  Improved  ship  motion  modeling  for  mathematical  modeling  of  organic  air  asset 
availability 

Continued  progress  was  also  made  in  creating  efficient  models  to  assess  the  likelihood  of  operating  organic 
rotorcraft  from  a  monohull  ship  under  varying  operating  conditions  such  as  ship  motion,  low  visibility  & 
poor  visual  cues,  restricted  landing  areas,  airwake  turbulence  and  helicopter  motion,  A  dynamic  interface 
simulation  environment  was  developed  that  uses  ship  motion  time  histories  to  provide  a  probabilistic 
analysis  of  organic  air  asset  availability  [1],  It  also  uses  a  quantification  of  pilot  workload  using  standard 
deviations  of  control  stick  position  and  velocity  to  quantify  a  Workload  Rating  (WR).  An  overview  is 
provided  in  Figure  6. 


Dynamic  Interface  Simulation  Environment 


Turbulence 


* 


Ship-Helicopter  Dynamic  Interface  Simulation 
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Figure  6.  SimuLink  Dynamic  Interface  Simulation  Environment 
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An  example  of  a  ship  motion  time  history  from  Ref.  1  for  a  TMV  Fast  Ferry  in  the  heave  direction  can  be 
seen  in  Figure  7. 


Figure  7 .  TMV  114  Fast  Ferry  Heave  Ship  Motion  Time  History 

Ship  motion  time  histories  are  needed  in  order  to  run  the  dynamic  interface  simulation  environment  for  a 
given  set  of  simulation  conditions  such  as  ship  design  variables,  sea  state,  pilot  delay,  etc.  From  the 
developed  time  histories,  a  random  sampling  technique  can  be  used  as  a  simulation  input  to  determine  if  a 
helicopter  can  land  on  the  ship  deck  during  the  simulation  time  window. 
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Figure  8.  Technique  for  Random  Sampling  of  Ship  Motion  Time  Histories 

Based  on  these  values  obtained  at  each  time  window,  it  becomes  possible  to  determine  the  pilot  workload 
rating  and  the  maximum  tracking  error  that  occurs  as  the  pilot  attempts  to  land  the  helicopter.  This  can  be 
quantified  as  a  Mission  Success  Rating  (MSR)  percentage  [1].  An  Artificial  Neural  Network  (ANN) 
regression  is  used.  The  ANN  is  a  feed- forward  design  with  two  hidden  layers  that  uses  sigmoidal/linear 
basis  functions.  The  output  from  the  ANN  is  a  prediction  of  the  landing  success  rate. 


Figure  9.  Artificial  Neural  Network  Regression  for  Determining  Mission  Success  Rate 
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The  following  figure  is  an  example  of  the  output  that  results  from  the  simulation  results.  The  helicopter 
used  in  this  simulation  is  a  BO- 105  helicopter.  The  ship  motion  histories  used  are  from  the  TMV  1 1 4  Fast 
Ferry.  Additionally,  the  pilot  model  used  is  an  optimal  control  model  with  a  response  delay  of  0,2  seconds 


MSR  beta=fl  _ j  _ _ j 

Us  1234S67B9  10  1112 


Figure  10.  Mission  Success  Ratings  for  Helicopter  Landing 


The  MSR  values  displayed  in  each  of  the  colored  boxes  are  for  varying  levels  of  significant  wave  height 
(Hs)  and  zero  upcrossing  period  (Tz),  while  beta  indicates  the  direction  of  seas.  The  following  figure 
provides  an  example  graphical  display  of  the  ANN  surrogate  model  results. 


15 


L*vjh  i,mj 


Conditions: 

■  80-105  Helicopter 

■  0  degree  heading 

■  Landing  pad -80% 
length,  center  line 

■  Sea  state:  2m 
significant  wave  height, 
10  second  zero  crossing 
frequency 

■  GMt  (transverse  metric 
height}  -  2.7  meters 

■  Tn  (roll  natural  period) 
-4.5  seconds 


Figure  11.  Sample  ANN  Surrogate  Modeling  Results 

Initially,  simplified  hydrodynamics  were  used  to  obtain  the  ship  motion.  This  method  simplified  the  shape 
of  the  hull  to  a  rectangular  prism.  During  this  period  of  the  effort  the  focus  was  on  improving  the  modeling 
of  the  ship  motion,  since  the  d>namic  interface  between  the  ship  environment  and  the  helicopter  is  vital  to 
correctly  developing  the  proper  modeling  and  simulation  tools.  There  was  collaboration  with  Virginia  Tech 
to  utilize  existing  modeling  &  simulation  tools  to  obtain  power  spectral  density  (PSD)  plots  of  ship  motion 
for  various  monohull  surface  sh  ips.  A  PSD  describes  how  the  power  of  a  signal  or  time  series  is  distributed 
over  different  frequencies.  A  PSD  can  be  described  using  Equation  I : 


(1)  ®M-|»M|’sW 

Where: 

•  d>(w):  Power  Spectral  Density  (PSD)  of  ship  motions 

•  H(w):  Frequency  response  function 

•  S(w):  PSD  of  sea  waves 

•  Inputs  for  heave  and  pi:ch  response  functions 

•  Draft,  length,  breadth,  Froude  number,  heading,  box  coefficient,  etc. 

•  Inputs  for  roll  response  function 
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*  Draft,  length,  breadth,  Froude  number,  heading,  box  coefficient,  transverse  met  centric 
height,  roil  time  constant,  etc* 

•  Sea  wave  model:  JONSWAP  spectrum  model  (Joint  North  Sea  Wave  Observation  Project)*  See 
Ref.  [2],  Sea  states  are  characterized  by  frequencies  and  amplitudes  of  waves. 

Example  PSD’s  of  the  heave,  roll  and  pitch  motions  fora  TMV  1 14  Fast  Ferry  are  provided  in  the  following 
figures  from  Ref.  [2]: 


Figure  12*  TMV  114  Fast  Ferry  Heave  PSD 


Figure  13,  TMV  114  Fast  Ferry  Roll  PSD 
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Figure  14,  TMV  114  Fast  Ferry  Pitch  PSD 

Figure  13  illustrates  the  goal,  namely  to  identify  a  technique  to  translate  PSD  plots  into  ship  motion  time 
histories. 


Figure  15,  TMV  114  Fast  Ferry  Pitch  PSD 

A  technique  for  generating  the  time  histories  so  as  to  simulate  the  ship  motions  over  time  was  identified  in 
Ref.  [3],  In  summary,  the  first  step  is  to  integrate  the  PSD  over  frequency.  Then,  one  can  select  frequencies 

and  r^?as  follows: 


(2) 
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(3) 


Vn  CO 

1^(5)*  =  0.99 

0  0 


Now,  ^2*"‘>^#*-ican  be  equally  distributed  between  °*[  and  c°n ,  Amplitudes  at  the  select  frequencies 
are  obtained  as  follows: 


ak  = 


(  t*k  ^ 

2  jon,(4fc 

k  "*-l 


1/2 


,  £  =  1,2,— 


(4) 


Equations  (2)-{4)  imply  that  the  PSDs  calculated  in  Eq.  (1)  are  represented  as  the  sum  of  n  sinusoidal 
signals,  whose  amplitudes  are  condensed  so  that  the  frequency  distribution  over  power  closely  match  that 
of  the  integrated  PSD,  However,  it  should  be  noted  that  information  on  the  relative  phase  of  each  motion's 
time  history  is  lost  with  this  technique.  In  the  end,  though,  one  can  obtain  heave  (hs),  pitch  (0s),  and  roll 
(t^s)  of  the  ship  in  the  form  of  time  series.  In  this  analysis,  motion  variables  of  a  ship  follow  the  definitions 
and  sign  conventions  given  in  Ref  0. 


Focus  Area  3:  Operational  Modeling 

The  future  OPV  will  be  required  to  carry  out  a  number  of  Naval  tasks,  including  search  and  rescue,  maritime 
interdiction,  conduct  embargos  and  blockades,  engage  in  Anti -Surface  and  Anti-Aircraft  Warfare  (ASuW 
&  AAW),  perform  fishery  patrols,  provide  non-combatant  evacuation  (NEO)  and  even  anti-piracy,  to  name 
a  few.  In  order  to  perform  a  capabilities-based  analysis,  collaboration  with  other  entities  such  as  Naval 
Postgraduate  School  and  personnel  from  OSN,  CETENA  and  the  University  of  Genoa  was  necessary  to 
create  the  necessary  operational  models  to  assess  OPV  capability.  These  models  were  then  incorporated 
into  the  graphical  tradeoff  environment  using  surrogate  modeling  techniques  so  that  real-time  analysis  and 
exploration  of  the  entire  design  space  could  be  conducted.  Surrogate  modeling  enables  rapid  manipulation 
of  any  modeling  and  simulation  tools.  This  allows  highly  accurate,  equation-based  regressions  of  complex 
codes  with  negligible  (but  measureable)  loss  in  accuracy  of  the  original  tools.  The  surrogate  model  can  then 
be  executed  in  fractions  of  a  second  instead  of  hours  or  days. 
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input  Variables  and  ranges 
for  problem  of  interest 


Responses 


Surrogate  Model: 
{Created  through  Regression) 


Figure  16.  Surrogate  Modeling  Based  Approach  for  Operations  Model  Integration 

Traditional  Operations  Research/Ana  lysis  (OR/OA)  approaches  rely  on  simple  analytical  formulations  for 
mapping  the  measures  of  effectiveness  of  a  system.  As  the  missions  and  tasks  modeled  become  more 
complex,  models  that  capture  these  complexities  must  be  developed,  or  the  modeler  must  simplify  and 
abstract  the  situation  to  create  useful  and  credible  models  that  will  support  decision  making.  Traditional 
OR/OA  models  may  not  suffice,  and  over  time  have  become  more  complex,  e.g.,  using  numerical 
optimization,  discrete  event  simulation,  and  agent-based  modeling  and  simulation  (ABM&S).  These 
techniques  are  newer,  more  complex,  and  demanding  in  terms  of  implementation  and  evaluation.  In  order 
to  aid  the  decision  maker  during  design  space  exploration,  different  statistical  analysis  techniques  were 
investigated  for  their  applicability  in  this  regard.  The  first  technique  focused  on  investigating  statistical 
correlations  between  different  levels  in  the  hierarchy  that  consists  of  several  mappings  from  geopolitical 
goals  to  military  missions  and  naval  tasks  down  through  ship  systems  previously  depicted  in  Figure  2.  The 
goal  is  to  capture  hidden  trades  early  on  in  the  design  process  with  varying  ship  capability  levels  used  as 
an  input.  This  can  be  seen  in  the  following  figure. 
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Figure  17.  Statistical  Analysis  of  Ship  Capability  Level  vs.  Performance 

As  seen  in  Figure  17,  there  are  clear  trends  exhibited  in  the  design  space  that  may  be  exploited.  However, 
to  make  this  information  more  useful  to  the  decision  maker,  additional  investigation  into  automating  the 
statistical  analysis  for  various  analytical  objectives  must  be  investigated  and  refined. 


Lastly,  techniques  were  explored  for  formalizing  the  implementation  and  improving  the  use  of  Agent  Based 
Models  (ABM).  Traditional  ABM  applications  use  many  instantiations  of  a  simple  agent  to  study  aggregate 
behaviors  that  are  difficult  to  predict  from  the  study  of  the  agents  in  isolation.  Rather  than  being  a 
completely  scientific  exercise,  the  creation  and  execution  of  agent-based  models  still  retains  elements  of 
art.  Anti-piracy  was  determined  to  be  a  good  candidate  for  testing  cue  to  the  interactions  that  can  occur 
between  agents  and  the  relevancy  to  OPV  design  requirements.  Figure  18  demonstrates  the  number  of 
recorded  pirate  attacks  against  commercial  vessels  off  the  coast  o:  Somalia  in  2009  [S].  Figure  19  also 
shows  an  overlay  of  an  internationally  recommended  transit  corridor  (1RTC). 
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Figure  18. 2009  Somali  Pirating  Attacks 


Figure  19.  Internationally  Recommended  Transit  Corridor  (tRTC). 

An  ABM  was  created  using  the  NetLogo  M&S  tool.  The  ABM  scenario  was  modeled  with  various  numbers 
of  tankers  and  containe*  ships  transiting  the  IRTC.  An  OPV  and  helicopter  can  then  be  simu.ated  as  agents 
patrolling  the  IRTC  to  disrupt  and  deter  pirate  attacks.  An  additional  layer  of  complexity  is  added  by  the 
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presence  of  fishing  vessels  that  can  be  used  by  pirates  to  mask  their  approach.  Figure  20  gives  a  graphical 
overview  of  the  ABM. 


Figure  20.  NetLogo  Agent  Based  Anti-Piracy  Model 

A  Measure  of  Effectiveness  (MOE)  was  developed  in  order  to  quantify  the  impact  of  the  OPV.  The  MOE 
is  shown  in  Equation  5: 


(5) 


f 

MOE  -  I 

V 


Number  Intercepted  +  Number  Escaped 
Number  Pirates 


Y 

A 


Number  Attacks 
Number  Pirates 


\ 


The  simulation  uses  the  following  criteria  in  determining  the  MOE  for  each  engagement. 


•  Approaches:  Visual  contact  has  been  made  between  the  civilian  cargo  ship  and  the  pirates 
■  Attacks:  The  pirate  vessel  catches  the  civilian  ship  and  attempts  to  board 

*  Successes:  The  pirates  successfully  take  the  ship 
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•  Escapes:  The  pirate  vessel  is  scared  off  by  the  patrol  ship 

*  Number  Pirates:  The  total  number  of  pirates  during  the  scenario 

Figure  21  provides  an  accounting  of  the  29  factors  that  can  be  varied  in  the  execution  of  the  model.  Ranges 
for  each  variable  were  estimated  using  open  source  information. 
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Figure  21.  Anti-Piracy  Model  Factors 
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This  represents  a  complex  model  with  many  inputs  that  affect  the  interaction  of  the  different  agents  within 
the  simulation.  Modeling  and  simulation  environments  that  are  stochastic  in  nature  present  a  multitude  of 
problems  in  the  creation  of  surrogate  or  meta -models,  notably  the  reduction  of  the  quality  of  fit  due  to 
statistical  error.  The  first  issue  is  in  determining  the  minimum  number  of  repetitions  required  for  interval 
estimates  to  hold  true  regardless  of  interval  width.  These  intervals  are  then  used  for  meta-model  regressions. 
Four  measures  were  examined.  Sample  mean  and  variance  interval  coverage  accuracy  was  studied  by 
varying  skewness,  kurtosis,  desired  coverage,  and  sample  size  within  the  Pearson  family  of  distributions. 
Binomial  proportion  and  quantile  interval  coverage  accuracy  was  studied  with  the  standard  normal  with 
varying  sample  size,  desired  coverage,  and  quantile  level.  Finally,  heuristic  measures  to  determine  how 
repetitions  are  related  to  the  quality  of  the  meta-model  fit  were  developed  based  on  the  experimentation 
with  a  canonical  problem.  The  ratio  of  confidence  interval  width  to  the  range  of  sample  measures  was  found 
to  be  an  indicator  of  the  impact  of  statistical  error  on  the  quality  of  model  fit.  Regression  methods  of 
weighted  least  squares  (WLS),  ordinary  least  squares  for  constant  sample  sizes,  and  constant  interval  widths 
were  compared.  The  WLS  method  is  suggested  for  stochastic  regressions  of  simulations.  A  detailed 
explanation  of  the  methodology  can  be  found  in  Ref.  [6],  Based  on  this  methodology,  40  repetitions  were 
determined  to  be  an  adequate  number.  A  design  of  experiments  (DoE)  was  then  constructed  in  order  to  aid 
in  constructing  the  surrogate  model,  A  summary  is  provided  here: 

■  The  DoE  used  was  a  Latin  Hypercube  Sampling  (LHS)  DoE 

■  290  LHS  points  were  created 

*  1 0  per  factor  (29  factors  examined) 

■  Maximum  Correlation:  0,0043 

•  Model  runtime:  720  Hr  (30  Days) 

■  Repetitions:  40 

*  Shown  by  past  research  that  this  is  more  than  sufficient  for  mean  estimates  used  in 
conjunction  with  weighted  linear  regressions  weighted  on  inverse  of  confidence  interval 

■  2  DOE  sets  (with  helicopter  and  without  helicopter) 

The  results  of  operations  of  the  OPV  both  with  and  without  the  helicopter  were  then  analyzed.  For  the  case 
of  the  OPV  operating  with  the  helicopter,  the  OPV/helicopler  patrol  pattern  used  is  shown  in  Figure  22. 
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Figure  22.  Combined  OPV/Heli copter  Search  Pattern 
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Figure  23  provides  the  results  of  the  surrogate  model  regression  quality  of  fits  for  the  OPV/Helieopter  patrol 
combination*  Figure  24  depicts  the  influence  of  different  variables. 
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Figure  23*  OPV/Helicopter  Actual  by  Predicted  and  Residual  by  Predicted  Plots  for  Surrogate  Model 
Goodness  of  Fit 
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Figure  24.  OPV/Helicopter  Parameter  Estimates 


The  results  of  the  analysis  show  that  an  R-squared  value  of  0.92  and  the  scattering  pattern  of  the  residuals 
provide  an  adequate  surrogate  model  fit.  Also,  the  most  influential  variable  is  the  maximum  speed  of  the 
pirate  boats.  This  is  intuitive  since  the  faster  the  pirates  travel  the  easier  it  is  for  them  to  overtake  the  cargo 
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ships.  The  analysis  also  shows  that  a  significant  portion  of  the  response  is  controllable.  This  includes  the 
following: 


*  Operations 

*  Reduce  vehicle  search  area  More  vehicles  in  the  area 

■  Performance 

■  Improve  identification  of  pirates 

■  Keeps  eyes  in  the  sky 

■  Increase  OPV  max  speed 

■  Increase  OPV  cruise  speed 

The  following  figures  are  the  results  from  running  the  simulation  when  there  is  no  helicopter  available  for 
use  by  the  OPV,  Figure  25  provides  the  results  of  the  surrogate  model  regression  quality  of  fits  while  Figure 
26  depicts  the  influence  of  different  variables. 


MOE_Mean  Predicted  P<  0001 
RSq=0.93  RMSE==0.2551 


MOE  Mean  Predicted 


Figure  25,  OPV  without  Helicopter  Actual  by  Predicted  and  Residual  by  Predicted  Plots  for  Surrogate 
Model  Goodness  of  Fit 
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Sorted  Parameter  Estimates 
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Figure  26.  DRV  without  Helicopter  Parameter  Estimates 

The  results  of  this  analysis  show  that  an  R-squared  value  of  0,93  and  the  scattering  pattern  of  the  residuals 
provide  an  adequate  surrogate  model  fit.  Also,  the  most  influential  variable  is  the  maximum  speed  of  the 
pirate  boats.  This  was  also  seen  in  the  scenario  when  the  OPV  was  patrolling  with  the  helicopter.  The 
analysis  also  shows  that  a  significant  portion  of  the  response  is  also  controllable  in  this  case.  This  includes 
the  following: 


■  Operations 

*  Reduce  vehicle  search  area  ->  More  vehicles  in  the  area 

■  Performance 

■  Improve  identification  of  pirates 

■  Increase  OPV  cruise  speed 

■  Note:  OPV  Max  speed  not  as  important 

In  contrast  to  what  was  observed  in  the  case  of  the  OPV  operating  with  the  helicopter,  OPV  max  speed 
plays  less  of  role  in  mission  success.  This  analysis  has  determined  that  in  situations  where  the  OPV  will  be 
patrolling  without  a  helicopter  OPV  cruise  speed  is  more  important.  OPV  maximum  speed  gains  increased 
importance  when  operating  with  a  helicopter  since  the  OPV  must  be  able  to  quickly  relocate  to  an  area 
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designated  by  tlie  forward  patrolling  helicopter  in  order  to  disrupt  pilot  activities.  Overall,  impelling  the 
results  of  this  ABM  allows  decision  makers  to  estimate  the  impact  of  the  systems  level  design  all  the  way 
up  to  tactical  and  even  higher-level  strategic  goals.  This  also  requires  the  continued  study  of  complex 
modeling  and  simulation  Techniques  such  as  ABM  in  order  to  more  efficiently  design,  use,  and  integrate 
them  into  the  decision  making  process. 
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BMD  Wargaming  Task 

The  main  objective  of  the  BMD  task  was  to  adapt  Georgia  Tech’s  Aerospace  Systems  Design  Laboratory 
(A SDL)  advanced  design  techniques  to  facilitate  the  construction  of  an  analytical  environment  capable  of 
performing  real-time  trades  of  interest  to  decision  makers  leveraging  the  analytical  collaboration  facilities 
of  ASDL,  Many  of  the  advanced  design  methods  and  know-how  that  were  applied  for  this  project  have 
been  developed  through  other  QNR-funded  ASDL  projects.  The  Ballistic  Missile  Defense  Phased  Adaptive 
Approach  (BMD  PAA)  (renamed  HDBAS  -  Homeland  Defense  using  the  Ballistic  Missile  Defense  System 
against  Southwest  Asia)  is  a  U.S.  Northern  Command  (USNGRTHCOM)  Warfighter  Challenge  within  the 
FY1 1  Joint  Concept  Development  and  Experimentation  Campaign  Plan,  conducted  by  the  Joint  Staff-led 
Joint  and  Coal  ition  Warfighting  (JCW)  group  (formerly  known  as  USJFCOM  J7).  The  goal  of  this  effort 
is  to  leverage  the  USJFCOM  Warfighter  Challenge  as  case  study  to  better  understand  the  suitability  of  using 
surrogate  modeling  within  defense  modeling  and  simulation  applications.  It  is  hoped  that  a  better 
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understanding  of  the  specifics  surrounding  the  acceptable  uses  and  limitations  of  the  various  types  of 
advanced  methods  will  be  obtained. 

The  researchers  followed  a  regimented  approach  (see  Figure  1:  HDBAS  Work  Plan)  to  developing  the 
necessary  tools  to  support  the  HDBAS  wargame. 


Figure  1:  HDBAS  Work  Plan 

Step  1  -  Wargame  Requirements  Analysis 

The  first  step  in  the  proposed  process  is  a  thorough  analysis  of  the  problem  space.  The  wargame 
requirements  call  for  the  input  from  three  phases  of  analysis:  (1)  Development,  (2}  Assessment,  and  (3) 
Solutions.  The  Development  Phase  will  include  single  threat  scenarios.  The  Assessment  Phase  will  include 
raids  as  well  as  more  detailed  single  threat  analysis.  The  objective  of  this  phase  is  to  identify  gaps  and 
vulnerabilities  in  the  proposed  PAA.  The  Solutions  Phase  will  address  these  gaps  and  vulnerabilities  that 
have  been  identified.  Steps  3-7  will  be  repeated  for  each  phase.  Step  8  will  include  the  analytical  GUI  of 
all  three  phases.  Requirements  for  the  types  of  trade  studies  the  decision  makers  would  like  to  perform 
during  the  analytic  wa  rgame,  the  necessary  "dials"  that  the  analyses  should  have  available  for  adjustment 
to  support  such  trades,  and  the  types  of  performance  metrics  the  decision-makers  are  interested  in  seeing 
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will  be  developed  and  stated  explicitly.  This  step  is  critical  because  it  drives  the  requirements  for  all 
subsequent  steps. 

Step  2  -  Scenario  Definition 

During  the  scenario  definition  step  the  team  will  produce  a  detailed  description  of  the  range  of  scenarios 
to  be  modeled  in  order  to  support  the  analytical  wargame.  A  definition  of  available  blue  forces, 
constraints  on  force  apportionment  and  deployment,  list  and  prioritization  of  defended  assets,  threat 
epoch  characteristics,  and  possible  launch  locations  and  trajectories  will  be  agreed  upon. 

Step  3  -  Construction  ofM&S  Environment 

This  step  will  focus  on  identifying  and  constructing  a  modeling  and  simulation  environment  suitable  for 
generating  the  data  required  to  create  the  surrogate  models  needed  to  support  the  analytical  wargame. 
Existing  tools  with  extensive  support,  such  as  SAIC-owned  WILMA,  could  most  likely  be  setup  and  run  very 
quickly,  but  contractor  time  might  be  expensive.  Other  existing  tools,  like  EADSIM,  might  require  climbing 
a  steep  learning  curve,  and  take  longer  to  get  setup  and  calibrated  for  this  problem,  but  offer  the  ability 
to  run  and  rerun  cases  as  needed  at  no  additional  cost.  A  detailed  M&S  tradeoff  will  need  to  be  performed 
to  settle  upon  the  best  overall  compromise. 

Step  4  -  Identify  Variables  of  Interest 

The  trades  identified  during  the  wargame  requirements  analysis  are  mapped  to  specific  variables  available 
in  the  created  M&S  environment.  Appropriate  ranges  for  the  variables  are  identified  according  to  the 
scenario  definitions  obtained  in  step  2.  The  performance  metrics  (responses)  are  likewise  mapped  to 
specific  output  variables  available  in  the  created  M&S  environment.  An  analysis  plan  must  then  be 
created  to  determine  which  performance  metrics  will  need  to  be  modeled  using  surrogate  models  and 
which,  if  any,  can  be  quickly  calculated  based  on  the  output  of  said  surrogates. 

Step  S  -  Create  Design  of  Experiments  (DoE) 

Subject  to  the  number  of  identified  input  variables,  and  restrictions  on  the  number  of  simulations  or  the 
allowed  total  runtime,  a  DoE  is  constructed  to  ensure  that  the  simulation  runs  provide  the  greatest 
amount  of  statistically  significant  data  and  capture  the  variability  of  the  responses  to  the  greatest  degree 
possible.  This  step  provides  the  run  matrix  to  be  used  in  executing  the  simulations. 

Step  7  -  Construct  Surrogate  Models 

Various  non-linear  regression  techniques  will  be  employed  to  construct  a  transfer  function  that  relates 
the  identified  inputs  and  responses.  The  surrogate  models  give  the  analyst  the  ability  to  execute  data 
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calls  nearly  instantaneously  to  the  level  of  fidelity  of  the  underlying  M&S  tools  used  to  create  them.  This 
enables  the  use  of  real-time  probabilistic  analysis  techniques  that  were  previously  prohibited  by  the  long 
runtimes  of  the  BMD  M&S  tools. 

Step  8  -  Build  Wargame  GUI 

A  graphical  user  interface  (GUI)  will  be  created  to  leverage  the  power  of  the  surrogate  models  in  support 
of  the  analytical  wargame.  The  GUI  will  be  designed  such  that  it  is  able  to  visually  convey  the  proba  bilistic 
output  of  the  surrogate  model  analyses  in  a  meaningful  way,  while  at  the  same  time  enabling  real-time 
interaction  with  the  input  variables  and  underlying  assumptions.  The  GUI  will  make  full  use  of  the 
functionality  of  ASDL's  analytical  facilities,  the  CoVE  and  CoDE,  described  below. 

Step  9  -  Perform  Probabilistic  Studies 

Once  the  surrogate  models  have  been  created,  they  can  be  used  in  place  of  the  underlying  modeling  and 
simulation  tools  to  perform  probabilistic  analysis  and  design  studies  using  many  thousands  of  surrogate- 
model-generated  data  points.  The  goal  of  this  step  is  to  explore  the  decision  space  and  gain  an 
understanding  of  the  performance  of  the  BMD  system  being  studying  and  put  forth  recommendations 
about  how  to  maximize  its  performance. 

Step  10-  Conduct  Wargame 

The  final  step  of  the  proposed  approach  is  the  actual  conduct  of  the  analytical  wargame.  A5DL  is  preparing 
to  use  the  Secure  COVE  facility.  The  high  resolution  displays  and  ample  seating  are  ideal  for  allowing 
decision  makers  to  collaboratively  perform  real-time  trade  studies. 

ASDL  developed  and  tested  a  Rapid  Tradeoff  Visualization  Tool  (RTVT)  to  support  Development  Phase 
analysis.  Additionally,  ASDL  co-authored  a  detailed  17-page  whitepaper  entitled  'Understanding  the 
Metrics  -  A  Survey  of  Metrics  for  Use  in  SoS-Level  BMD  Decision  Making1  and  distributed  to  leadership. 
Because  of  the  complexity  of  the  BMD  system,  many  metrics  exist  to  describe  the  performance  of  the 
system  at  various  levels.  Particular  importance  is  given  to  the  distinction  between  single-threat  scenario 
performance  metrics  and  multiple-th  reat  raid  performance  metrics.  It  is  shown  that  single  threat  scenario 
metrics  such  as  Probability  of  Engagement  Success  (Pb)  and  Single  Shot  Probability  of  Kill  (Pssk)  are 
unacceptable  for  use  as  multiple-threat-scenario  metrics.  Probability  of  Zero  Leakers  (Pzl)  is  then 
presented  as  a  viable  multiple-shot-scenario  metric  and  its  limitations  explained.  Expected  Number  of 
Leakers  (E(L))  is  introduced  as  a  way  to  overcome  the  limitations  of  PZL  without  requiring  any  additional 
data.  Finally,  Raid  Protection  Level  (RPL)  and  Average  Raid  Protection  Level  (RPL)  are  presented  as  a  viable 
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way  of  describing  the  expected  defensibility  of  a  particular  point  within  a  raid.  Examples  are  provided 
throughout  to  show  that  incorrect  selection  of  scenario-level  metrics  can  drastically  change  the  outcome 
of  the  study.  This  whitepaper  has  been  supported  by  MDA,  NORTHCOM,  JCW  and  SAIC  The  whitepa  per 
was  endorsed  by  senior  leadership  (including  Lt.  Gen.  Obering)  and  a  draft  is  included  in  the  appendix. 

Run  Matrix  Design 

Run  Matrix  size  was  a  significant  issue  that  needed  to  be  addressed 

■  Evaluating  all  possible  combinations  within  the  trade  space  is  very  computationally  expensive 

■  The  million  combinations  represented  by  the  variables  in  the  table  to  the  right  would  take 
4160  hours  to  run  at  1  minute/case  on  24  processors 

■  Not  all  combinations  within  the  trade  space  are  of  interest,  so  evaluating  all  possible  combinations  is 
not  even  desirable 


A  method  is  needed  for  intelligently  selecting  which  combinations  to  evaluate,  while  simultaneously 
reducing  the  Run  Matrix  size 


Previous  approach  relied  on  a  "smart"  design,  reducing  the  number  of  cases  by  manually  pruning  the  run 
matrix 

*  Only  variable  combinations  that  are  directly  relevant  to  the  study  are  evaluated 

*  This  approach  produces  an  order  of  magnitude  reduction  in  the  required  cases 
■  Results  are  biased  by  running  only  those  cases  which  are  seen  as  most  likely 

-  Statistical  significance  of  the  results  is  lost  as  branches  of  the  trade  space  are  pruned 

ASDL  introduced  Design  of  Experiments  (DOE)  into  the  Run  Matrix  Design  approach 
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■  Enables  order  of  magnitude  reduction  of  required  cases,  similar  to  manual  pruning 

■  Maintains  an  uncorrelated  Run  Matrix,  maximizing  the  amount  of  information  gathered  with  the 
cases  evaluated 

ASDL  briefings  on  the  fundamentals  of  DOE  enabled  the  transition  from  the  old  approach 
"  Design  of  Experiments  (DOE)  is: 

*  Purposeful  changes  to  the  inputs  to  a  process  in  order  to  observe  the  corresponding 
changes  in  the  outputs 

■  Inputs  also  called  design  variables  or  factors 
m  Outputs  also  called  responses 

•  A  set  of  runs  designed  to  maximize  information  and  minimize  experimental  effort 

Key  Concepts  and  Terminology 

*  Correlation:  Statistically  induced  non-independence  of  Input  variables 

■  Orthogonality:  Implies  zero  correlation  between  experimental  factors 

■  Le*  A  purely  orthogonal  design  Is  used  to  maintain  the  independence  of  independent 
variables 

Resulting  Run  Matrix  Design  Options: 
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Full  Factorial 


-900,000 


Quantifies  performance  for  every 
possible  combination 


Computationally  expensive 
Evaluates  many  uni  tkety 
combinations 


Manual  Pruning  ~  166r000 


Reduced  fl  of  Cases  when  compared  to 
Full  Factorial 

Evaluates  all  cases  of  interest  and  only 
those  cases 


Cannot  answer  "what-if"  questions 
that  were  not  explicitly  stated  at  the 
start 

Results  become  biased  and 
statistical  significance  is  lost  as 
branches  of  the  trade  space  are 
pruned 


• 

Specified  cases  can  be  incorporated  into 
the  Run  Matrix  generation 

*  Does  not  provide  for  a  1-to-l 

Nearly 

-50,000 

• 

Can  answer  "what-if"  questions 

comparison  between  Blue  Force 

Orthogonal  Arrays 

* 

throughout  the  trade  space 

Provides  guidance  for  selection 
subsequent  study  cases 

architectures 

Taguehi  Arrays  ~  120,000 


Provides  a  1-to-l  comparison  between 
Blue  Force  architectures 
Cart  answer  "what-if"  questions 
throughout  the  trade  space 
Provides  guidance  for  selection 
subsequent  study  cases 


*  Less  e ff i Ci ent  than  O A  Q ption  1 

•  Additional  cases  needed  to  quantify 
the  impact  of  interactions 


Rapid  Tradeoff  Visualization  Tool 

The  data  analysis  for  this  project  leveraged  the  experience  of  ASDL  in  creating  interactive,  dynamic  data 
visualization  tools  /  GUI.  The  use  of  these  visual  displays,  in  conjunction  with  advanced  visual  analytics 
techniques  will  allow  the  analyst  to  view  the  data  in  bulk,  find  trends  and  patterns,  and  examine  the 
influence  of  assumptions  more  easily  than  traditional  approaches  to  data  analysis. 

Data  generated  in  each  phase  of  the  study  is  stored  within  a  database  structure.  With  the  volume  of  data 
expected,  this  ensured  intuitive  and  responsive  access  to  all  data  from  within  the  visualization  interface. 
Output  files  in  the  form  of  ASCII  text  files  will  be  imported  into  the  database  using  custom  written  scripts 
that  will  be  implemented  in  the  visualization  tool.  All  database  management  was  performed  within  the 
visualization  tool  framework.  This  enabled  all  post-processing  and  analysis  operations  to  be  performed 
on  a  single  system,  minimizing  data  transfers  and  streamlining  the  overall  process.  An  overview  of  the 
integrated  framework  is  shown  in  Figure  2. 
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Run  Matrix 
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Figure  2:  The  Integrated  B1VID  PAA  Rapid  Tradeoff  Visualization  Tool  Framework 

One  or  more  visualization  tools  with  customizable  layouts  were  created  to  support  the  analysis  of  the 
results  of  each  phase.  These  interfaces  allowed  for  the  study  of  relationships  among  the  assumptions, 
scenario  configurations,  and  BMDS  performance  metrics.  Central  to  the  visualizations  for  each  phase  is  a 
map  feature  with  the  ability  to  display  overlaid  data  in  both  two  dimensions  and  three  dimensions.  User 
interaction  with  the  maps  as  well  as  other  analysis  modules  in  the  visualization  interface  will  automatically 
query  the  database  of  simulation  inputs  and  outputs  to  display  the  geographic  differences  in  performance 
of  various  BMD  configurations.  This  tool  wilt  provide  the  user  with  the  ability  to  rapidly  sort  through 
uninteresting/non-limiting  cases  and  quickly  find  troublesome  areas  that  need  to  be  examined  in  the 
Assessment  Phase,  An  example  data  structure  is  presented  in  Figure  3. 
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Figure  3:  HDBAS  Rapid  Tradeoff  Visualization  Tool  Example  Modules  and  Data  Structure 


The  modules  for  the  RTVT  are  custom,  developed  for  use  for  HDBAS.  JMP  has  many  built  in  JMP  Scripting 
Language  (JSL)  functions  and  visualizations  but  this  was  a  fairly  large  development  effort.  The  current  list 
of  modules  is  as  follows: 

1.  Box  Plots:  In  the  JMP  dashboard,  a  box  plot  is  used  to  visualize  the  Probability  of  Engagement 
Success  (PES)  of  the  entire  BMDSof  defen  ding  a  specific  targeted  site  or  defending  against  a  specific  threat 
site.  The  individual  data  points  in  these  hypothetical  box  plots  represent  the  average  PES  for  a  single 
interceptor  shot  at  a  threat  missile  launched  from  a  specific  threat  site  to  a  specific  targeted  site  using  a 
unique  BMDS  architecture. 

2.  ScatterPlotMatrix:Annxnmatrixwith2Dcomparisonsofanycombinationof  user  selectedinput 
and  output  variables.  It  is  very  useful  for  constraint  analysis,  inverse  design,  and  to  find  trends  and 
correlations  with  large  sets  of  data.  Data  can  be  color  coded  by  discrete  scenario  variables  like  threat 
type,  launch  location,  etc. 
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3.  Heat  Map:  The  heat  map  module  is  responsible  for  analyzing  the  probability  of  engagement 
success  (PES)  for  asset  architectures  and  displaying  the  information  as  a  series  of  colors  which  identify 
relative  value  of  PES  for  threat-aim  site  pairings.  The  heat  map  analysis  presents  data  in  the  form  of  a 
series  of  (n  x  m)  matrices  where  the  rows  and  columns  display  the  launch  and  aim  points  included  in  the 
given  study.  Each  matrix  of  launch-aim  points  identifies  a  single  BMDS  asset  architecture  [Le.  radar 
locations  and  types  as  well  as  interceptor  type).  The  user  wifi  be  able  to  select  what  threat  architecture 
they  would  like  to  view  via  drop  down  menu. 

4.  Engagement  Sequence  Group  (ESG)  Ana  lysis:  The  purpose  of  a  n  ESG  analysis  is  to  determine  what 
the  placement  of  assets  was  during  a  specific  defense  sequence.  This  allows  the  determination  as  to  what 
assets  was  the  best  performing  and  at  what  locations  they  performed  best  at.  This  allows  the  person  to 
make  intelligent  trades  in  order  to  ensure  that  the  best  assets  (Radars,  Space  based  assets  ...}  are  placed 
at  the  best  locations  performing  appropriate  roles  in  the  ESG  (such  as  Commit,  Track,  Update,)  such  that 
the  desired  PES  can  be  achieved.  To  that  end,  a  simple  visualization  was  needed  to  quickly  see  which  roles 
were  being  filled  and  what  assets  filled  those  roles  with  different  scenarios.  A  cell  plot  function  was  used 
for  the  visualization. 

5.  Principal  Component  Analysis:  Principal  Component  Analysis  (PCA)  is  a  method  used  to  reduce 
the  dimensionality  of  a  set  of  data.  One  application  of  this  method  is  to  define  the  overall  variability  of  a 
set  of  data.  When  confronted  with  large  sums  of  data  that  happens  to  be  stochastic,  this  method  could 
aid  in  the  determination  of  how  many  repetitions  are  required  to  obtain  the  desired  confidence  intervals. 
Another  application  is  the  visual  display  of  the  data.  By  reducing  the  dimensionality  one  can  better 
understand  the  behaviors.  For  example  a  15  dimensional  space  could  be  represented  as  a  3  dimensional 
space  while  observing  the  majority  of  the  variability  of  the  space.  However,  the  transformations  made  can 
result  in  very  confusing  metrics.  Another  application  is  to  use  PCA  to  aid  in  selection  Measure  of 
Effectiveness  or  Performance  of  a  system.  This  may  not  happen  however,  because  variables  that  need  to 
be  maximized  and  minimized  could  be  correlated  along  the  same  principal  factor.  This  method  could  be 
used  to  aid  other  metrics  that  are  limited  by  the  number  to  dimensions  that  can  be  used.  Examples  of  this 
are  Partition  Analysts  and  Heat  Map, 

6.  Partition  Analysis:  Partition  Analysis  is  a  technique  often  used  as  a  data  mining  tool.  PA  is  good 
for  finding  relationships  within  the  data  without  having  a  defined  model.  In  addition,  PA  is  useful  when 
dealing  with  large  amounts  of  data,  in  general  a  PA  will  determine  the  factors  that  control  the  greatest 
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variability  on  the  response.  This  can  be  useful  for  determining  which  factors  to  investigate  further.  In  this 
application,  the  PA  will  be  utilized  in  determining  the  main  drivers  of  launch  and  aim  points.  This  analysis 
may  result  in  the  identification  of  critical  assets  or  even  launch  windows,  In  addition,  PA  could  be  used  to 
determine  which  aim  points  have  sufficiently  similar  PES  under  defined  conditions  that  they  could  be 
evaluated  as  nearly  identical.  PA  could  also  be  used  to  break  up  the  space  as  a  series  of  launch  to  target 
"corridors”  to  determine  which  routes  are  able  to  produce  the  best  radar  track  of  threats. 

7.  Cluster  Analysis:  Cluster  Analysis  (CA)  is  the  process  to  dividing  a  set  of  data  containing  n  pieces 
of  data  into  k  sets  of  data.  .  CA  can  be  used  to  help  identify  key  elements  or  patterns  in  the  data.  An 
example  would  be  to  cluster  by  probability  of  kill,  which  will  show  similarities  in  cases  that  may  have 
completely  different  launch  and  target  sites.  This  allows  the  analyst  to  compare  and  evaluate  variables 
that  may  have  a  considerable  impact  on  probability  of  kill  regardless  of  the  launch  or  target  site.  Another 
foreseeable  use  of  cluster  analysis  would  be  to  cluster  based  upon  the  time  the  threat  is  identified.  This 
analysis  would  allow  for  the  identification  of  areas  or  target  sites  where  tracking  the  threat  early  on  in  the 
engagement  is  difficult.  For  this  example,  the  relationship  between  the  time  of  identification  to  the 
probability  of  kill,  time  of  intercept,  and  radar  track  quality  could  easily  be  examined. 

8.  PES  versus  Intercept  Time:  A  PES  versus  intercept  time  plot  displays  exactly  what  its  name  claims, 
the  overall  system  PES  as  a  function  of  the  time  when  the  lethal  threat  was  intercepted.  In  the  case  of 
this  plot,  the  system  is  defined  as  a  single  interceptor  shot.  The  plot  is  split  into  multiple  windows  each 
showing  PES  curves  per  a  single  interceptor  launch  point.  Each  curve  represents  a  unique  system 
architecture.  Furthermore,  as  the  same  system  architecture  may  be  used  to  set  up  the  shot  from  each 
launch  site,  the  curves  are  color  coded  to  the  system  architecture.  This  allows  for  single  architecture 
performance  to  be  compared  across  launch  sites. 

9.  Engagement  Timeline:  An  engagement  timeline  analysis  is  useful  for  examining  a  series  of  events 
over  a  time.  Events  are  plotted  on  the  Y  axis  while  time  is  plotted  on  the  X  axis,  The  time  period  that  an 
event  that  occurs  appears  as  a  bar.  This  method  examines  one  engagement  sequence  group  at  a  time. 

10.  Analysis  of  Variance:  Analysis  of  Variance  (ANOVA)  is  a  statistical  modeling  method  in  which  the 
variance  of  data  is  clustered  into  sections  in  order  to  explain  the  sources  of  the  variation,  in  essence,  the 
variance  is  attributed  to  certain  variables,  which  gives  the  user  an  indication  to  which  variables  are  the 
strong  contributors  of  change. 
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1L  3D  Trajectory  Visualization:  This  displays  one  or  more  trajectories  over  a  3D  map  for  comparison. 
This  module  still  needs  a  lot  of  work  and  will  be  continued  after  the  project  ramps  up  in  April*  The 
additional  work  will  be  to  add  Meta  data  (such  as  launch  window  information,  radar  search  volumes,  etc,) 
to  the  curves. 


It  is  important  to  note  that  the  modules  can  be  used  independently,  but  when  used  together,  the  visual 
analytical  capability  is  greatly  improved.  An  example  would  be  to  looking  across  the  executed  trade 
space  to  determine  critical  factors  and  areas  of  improvement  (solutions)  that  would  have  the  biggest 
impact  across  this  space. 

The  analysis  environment  built  in  IMP  makes  a  very  large  number  of  analysis  approaches  and  usage  paths 
available.  The  sequence  in  which  each  of  the  modules  is  used  is  influenced  by  everything  from  the  specific 
questions  being  asked  to  the  analyst's  personal  preference  or  style.  It  is  likely  that  a  number  of  key 
approaches  will  be  identified  and  standardized  during  the  course  of  this  study.  However,  several  notional 
analysis  approaches  are  provided  below  to  serve  as  examples  of  the  flexibility  available  to  the  analyst.  An 
example  path  was  posed: 

1.  Observe  interesting  feature  on  Map  module;  use  the  mouse  to  select  the  cases  that  make  up 
that  feature  and  highlight  them  with  a  different  color 

2.  Plot  key  parameters  and  metrics  in  the  Scatter  Plot  Matrix  in  order  to  identify  unique  attributes 
of  the  cases  of  interest  which  show  up  with  the  highlight  color  applied 

3*  Use  filters  to  remove  unrelated  cases  from  being  plotted  in  the  modules 

4.  Examine  the  Trajectories  plot  to  see  if  any  additional  intuition  can  be  gained  based  on  the 
geometry  of  the  engagement,  the  radar  locations,  etc* 

5.  Use  the  Timeline  Decision  Aid,  Sattlespace  Plot  and  Parallel  Plot  to  compare  cases  and  better 
understand  the  effects  which  were  observed  at  the  beginning 

6.  Use  filters  to  bring  all  of  the  data  back  into  play,  and  use 
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Summary 


The  RTVT  was  created  using  methods  to  support  analysis  of  large  data  sets.  ASDL  experimented  with 
different  database  designs  and  efficient  ways  of  accessing  the  data  to  support  rapid  tradespace  analysis. 
Different  modules  were  created  so  thatthe  data  can  be  analyzed  in  bulk  and  in  different  groups  to  support 
performance  and  fault  analysis.  The  RTVT  was  used  during  the  Development  Phase  for  benchmarking  and 
for  assisting  with  the  Assessment  phase  run  matrix  design. 

Visual  Command  and  Control  Capability  Analysis  Tradeoff  Suite  (VC3ATS) 

The  Visual  Command  &  Control  Capability  Analysis  and  Tradeoff  Suite  (VC3ATS)  task  seeks  to  perform 
dynamic  Command  &  Control  (C2)  and  mission  analysis  with  Joint  Mission  Thread  (JMT)  architectures  for 
the  purpose  of  comparing  C2  alternatives  in  the  single  and  multi-mission  trade  space.  This  will  result  in  a 
formalized  methodology  that  will  produce  traceable!  defensible  results  while  analyzing  both  process  and 
materiel  alternatives* 

VC3ATS  utilizes  standardized,  static  Department  of  Defense  Architecture  Framework  (DoDAF)  products  as 
a  means  of  providing  JMT  generic  data  descriptions.  These  JMT  descriptions,  which  are  described  as  being 
at  the  "Tier  1”  level,  are  completely  reusable.  In  contrast,  DoDAF  products  developed  at  the  "Tier  2"  level, 
for  example,  are  much  more  specific  and  serve  as  subsets  to  Tier  1  information.  Furthermore,  the  use  of 
static  Tier  1  related  DoDAF  products  are  then  leveraged  to  allow  for  the  automatic  or  semi-automatic 
generation  of  C2  and  mission  analyses,  which  are  then  used  for  the  purpose  of  C2  portfolio  tradeoffs. 
Figure  1  provides  an  overview  of  the  structured  decomposition  analysis  framework  used  by  VC3ATS, 

The  successful  implementation  of  this  C2  analysis  framework  requires  the  development  of  specific 
methods  and  techniques  that  will  enhance  the  decision  makers'  ability  to  perform  meaningful 
comparisons  of  competing  C2  architectures.  Specifically,  VC3ATS  focuses  on  3  key  approaches  that  will 
enhance  the  analysis  of  military  C2.  These  approaches  include: 

1.  The  generation  of  feasible  architectures 

2.  Analyzing  C2  performance  and  mission  effectiveness 

3.  Making  C2  attributes  measurable  and  useful 

VC3ATS  provides  an  interactive  visual  environment  to  aid  in  the  generation  and  analysis  of  feasible 
architectu  res.  An  example  of  this  can  be  seen  in  Figure  1: 
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Figure  1.  Interactive  Visual  Environment  Screenshots 

The  current  analysis  framework  uses  the  operational  sequence  diagram  of  related  tasks/functions  as  the 
'work  horse1  of  the  analysis  framework.  Continued  progress  has  been  made  in  including  additional  DoDAF 
products  and  provided  data  sets  in  order  to  improve  the  generation  of  feasible  architectures.  Specifically, 
the  research  was  conducted  in  cooperation  with  the  Deputy  Director,  Command  and  Control  Integration 
(DD  C2K)  Joint  Staff  (16)  to  incorporate  additional  "Tier  2”  level  architecture  data.  Also,  the  scope  of  the 
analysis  was  broadened  with  the  inclusion  of  additional  JMT  data  so  that  both  single  and  multi-JMT 
analysis  could  be  conducted.  Further  development  of  the  analysis  framework  resulted  in  the  refinement 
of  the  analysis  methodology  to  include  both  top-down  and  bottom-up  perspectives.  User  sessions  were 
then  conducted  with  Joint  Staff  (J 6)  where  valuable  feedback  was  obtained  for  possible  future 
development  of  VC3ATS.  Finally,  objectives  for  the  next  phase  of  VC3ATS  development  were  identified, 
provided  that  funding  is  renewed  for  the  next  FY. 
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Previous  developmental  phases  of  VC3ATS  operated  with  limited  data  set  availability.  However,  VC3ATS 
is  now  at  a  critical  crossroads  where  more  data  is  becoming  available  at  the  same  time  as  VC3ATS 
analytical  capability  is  improving.  For  example,  through  collaboration  with  the  Deputy  Director,  Command 
and  Control  Integration  (DD  C2K)  Joint  Staff  (J6)  "Tier  2"  JMT  data  became  available.  A  description  of  the 
different  Tiers  can  be  seen  in  Figure  2. 


Tier  1  JMT 

High  level  generic  data  description  - 
totally  reusable  architecture- based 
info  sets. 


Tier  2  JMT  “Strands” 

Information  represents  specific 
documentation  required  to 
answer  a  particular  question  or 

solve  a  problem. 


Tier  3  JMT 

Systems  engineering  level  of  detail. 
Bit-level  analysis  with  enough  rigor  to 
inform  Test/Eval  and  Mod/Sim 
communities. 

Results  in  Coordinated  Implementation 


Figure  2.  Joint  Mission  Thread  Tiers 

Specifically,  Tier  2  data  was  obtained  for  the  Joint  Fire  Support,  Joint  Close  Air  Support  and  Joint  Personnel 
Recovery  missions.  This  enabled  the  multi-thread  analysis  capability  to  become  more  fully  developed. 
Below  is  a  summary  of  some  of  the  analysis  questions  pertaining  to  both  single  and  multi-thread  analyses. 

❖  Single  Mission  Thread  Analysis: 

•  What  processes/equipment  supports  the  JMT  best? 

■  What  DOTMLPF  changes  can  be  made  to  improve  performance? 

■  What  is  the  bottleneck/vulnerability  point  of  failure  in  the  process? 

■  What  C2  construct  enables  the  optimal  JMT  performance? 

❖  Multiple  Mission  Thread  Analysis: 

■  What  DOTMLPF  commonality  exists  across  all  JMTs? 
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What  systems,  functionality,  etc.  is  unique  across  the  threads? 

Where  are  the  overlaps  and  redundancies?  What  is  the  effect  of  eliminating  these? 
How  do  DOTMLPF  changes  in  one  thread  affect  the  performance  of  others? 


Simultaneously,  more  automated  data  ingest  ion  methods  were  identified,  making  it  easier  to  incorporate 
new  mission  threads  in  the  future.  This  intersection  also  allows  VC3ATS  to  be  rolled  out  for  analyst  use 
and  feedback  via  series  of  training  sessions.  These  sessions  were  conducted  and  valuable  feedback  was 
obtained.  The  following  list  highlights  a  sampling  of  the  feedback  obtained  from  the  user  sessions: 

*  Provide  ability  to  load  baseline  solutions  for  each  JMT/Architectureto  provide  user  with  a  starting 
point  for  comparisons 

—  How  is  the  jMT  currently  executed/what  are  the  current  systems  in  use  or  heavily  relied 
upon? 

1  Additional  system/program  data  would  be  useful  for  defining  solutions: 

-  Interoperability  between  systems  is  a  major  concern 
—  Lifecycle  data 

—  Budget  data 

-  Additional  metadata  about  systems,  missions,  etc. 

—  System  performance  (test  command  data)? 

*  Access  to  authoritative  data  sources  a  top  priority 

*  Refine  target  audience  list  -  Action  Item  for  J6 

—  For  e.g.,  how  will  Architects  who  generate  data  find  this  useful  when  VC3ATS  needs  all 
JMT  data  up  front  to  ingest 

*  Comparison  of  model  to  model  vs.  data  to  data 

—  Impact  of  change  to  one  model  has  on  other  model  may  be  lost/resident  only  in  person 
who  developed  models 

*  Add  a  reset/clear  option  for  solution  history 

*  Add  a  'Ctrl-21  button  to  undo  actions 

*  Add  ability  to  view  multiple  OV-5  &  SV-1  diagrams  as  separate  windows  (compare  changes  more 
easily) 

*  Auto-save  function 

*  Provide  catalog  of  user-selected  displays 

*  Scrollable  grid  lines  on  System-Function  Performance  Overview  (Bubble  Plot) 

*  Provide  ability  to  view  more  threads  simultaneously 

-  Speed  up  load  time  for  many  mission  threads 

-  Identify  which  displays  are  most  vital  to  load  initially,  then  load  others  as  needed 

*  Refine  system  comparison  list  (functions  unique/common  to)  and  make  clearer 

*  Clarify  mission  specific  views  vs.  global  views 
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Refinement  of  VC3AT5  Analysis  Perspectives 

Further  development  of  the  analysis  framework  resulted  in  the  refinement  of  the  analysis  methodology 
to  include  both  top-down  and  bottom-up  perspectives.  Sample  use  cases  were  developed  and  proved 
useful  in  the  Joint  Staff  User  Sessions.  For  the  Top-down  perspective,  the  following  was  assumed: 

Sample  Use  Case  1: 

■  Analyst  very  familiar  with  current  C2  systems 

*  Asked  to  support  analysis  for  a  recommendation  to  trim  down  number  of 
operational  C2  systems 

■  Must  recommend  systems  that  should  be  phased  out 

■  Must  also  be  sure  that  loss  of  these  systems  will  not  result  in  loss  of  operational 
capability 

■  Accomplished  by  identifying  common/redundant  functions  across  systems  and 
possible  substitutes 


Meanwhile,  for  the  Bottom-down  perspective,  the  following  was  assumed: 

Sample  Use  Case  2: 

■  A  new  commander  is  rotated  in 

■  Needs  to  understand  the  current  C2  systems  in  use  by  his  organization/service 
component 

a  Must  determine  if  other  options  exist  that  might  better  fill  needs  and  close 
capability  gaps 

■  However,  may  have  limited  knowledge  of  existing  systems 

■  Therefore,  needs  to  understand  both  the  uses  and  performance  of  current  and 
competing  systems 


Identification  of  Phase  4  Objectives: 

Funding  for  VC3  ATS  was  discontinued  prior  to  the  completion  of  the  next  phase.  The  following  objectives 
were  identified  for  the  subsequent  phase  of  VC3ATS  development,  provided  funding  can  be  renewed  at  a 
future  date: 

❖  Enable  DOTMLPF  trades  (Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel  and 
Facilities)  &  improved  traceability  for  acquisition  decision  makers 

■  Expand  across  the  entire  DOTMLPF  spectrum 

■  E.g.  Training  vs.  Force  Structure  Tradeoffs  or  Training  vs.  Technology  Tradeoffs 

❖  Continuetransitiontouser&incorporateuserfeedback 

■  Joint  Staff  User  Workshops  February  7, 2013 

❖  Feasible  Architecture  Development 
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1  Further  development  of  system  metrics  and  methodologies  for  developing  and  using 
those  metrics 

*  Incorporation  of  completed  Tier  (I  JMT  Data 
*1*  Complex  SoS  Architecture  (Solution)  Evaluation  &  Analysis 

■  Further  development  of  Business  Process  Mode!  Executable  Architecture  Platform 

*  Currently  limited  to  JPR  mission  and  older  version  of  the  BPM  £A 

*  New  data  available  in  the  Tier  It  JMTs  suggests  potential  to  investigate  expanding 
the  EA  for  VC3ATS 

■  Would  like  to  consider  both  an  expansion  of  the  use  of  current  Java-based  BPM 
approach  and  other  potential  EA  tools  for  phase  4 

Space  Mission  Architecture 

The  objective  of  this  task  is  to  demonstrate/  train,  and  transfer  Georgia  Tech  (GT)  Advanced  Design 
methodologies  to  the  NASA/MSFC  Advanced  Concepts  Office  (ACO)  in  order  to  enhance  NASA  design 
practices  and  enable  expanded  trade  study  capabilities*  This  effort  focuses  on  applications  for  in-space 
disciplinary  analyses,  Earth-to-orbit  (ETOJ  modeling,  and  physics  based  engine  modeling,  enabling  an 
encompassing  look  at  a  conceptual  design  for  Mars  exploration  architecture.  Opportunities  associated 
with  advanced  manufacturing,  materials,  in-situ  resource  utilization  (ISRLJ),  and  concepts  cf  operation 
(CONORS)  will  be  evaluated  utilizing  advanced  technology  impact  forecasting  methods.  The  challenge  of 
developing  a  fully  reusable  transportation  architecture  for  manned  Mars  missions  was  selected  in  order 
to  provide  a  demonstration  problem  of  current  relevance  to  NASA* 

The  technical  objective  of  the  architecting  study  is  the  identification  of  reusable  architectures  that  support 
manned  missions  to  Mars  and  its  moons.  Identified  architectures  should  describe  the  functional  and 
physical  elements  required  to  make  them  possible*  The  architecture  design  process  must  develop  how 
the  transportation  architecture  and  supporting  infrastructure  evolve  to  advance  from  today's  capabilities 
to  an  architecture  where  vehicles  can  operate  for  20  or  more  years,  with  a  stretch  goal  of  achieving  similar 
reusability  and  serviceability  to  what  is  experienced  with  aircraft  today. 

The  architecting  process  employed  in  this  task  infuses  object-oriented  simulation,  morphological  analysis, 
and  relational  databases  to  assemble  executable  models.  Components  and  technologies  comprise 
vehicles,  events  and  trajectory  segments  comprise  missions,  and  vehicles  closed  to  specific  missions 
comprise  architectures*  Because  the  technology  development  evolution  path  and  strategy  are  of  key 
interest,  an  additional  layer  is  added  to  the  problem  in  which  sequences  of  architectures  employed  over 
a  period  of  years  comprise  a  campaign.  Figures  1  shows  the  general  approach  and  current  dashboard 
development  strategy. 
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Figure  1.  General  Approach  to  Creating  Viable  Campaigns 
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Figure  2.  Dashboard  Tool  Oitline 

Various  chal  enges  are  incorporated  into  the  overa  I  setup  and  analysis  of  the  problem .  In-space  assets 
should  have  minimal  dependence  on  Earth  for  resupply  except  for  provision  of  food  and  crew  supplies. 
Except  for  the  launch  and  crew  Earth  entry  vehicles,  elements  are  not  ret  jrned  to  Earth  surface  for 
refurbishment.  Refurbishment  and  resupply  of  Mars-vidnity  vehicles  may  be  provided  In  orbit  or  on  the 
Mars  surface.  The  emplacement  of  any  resources  and  nfrastructure  used  ir  the  architecture  must  be 
accounted  for,  and  may  include  a  combination  of  government  ard  commercial  capabilities.  Identify 
capabilities  that  need  to  be  developed  and  provide  a  roadmap  of  how  those  capabilities  w>ll  be  introduced 
over  time  to  evolve  the  architecture. 
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To  date,  progress  has  been  made  on  a  number  of  fronts,  including  architecture  trade  space  definition, 
architecture  evolution  modeling,  and  space  transportation  system  sizing.  The  architecture  trade  space 
has  been  defined  via  a  novel  dynamic  matrix  of  alternatives  (DYMOA),  which  was  necessary  for  concisely 
capturing  all  of  the  possible  alternatives  in  a  system-of-systems  problem.  Architecture  evolution, 
interpreted  as  a  time  varying  problem  that  traces  the  path  of  development  program  investments,  has 
been  modeled  using  a  discrete  event  simulation  (DES).  An  integer  programming  routine  has  been 
wrapped  around  the  DES  in  order  to  optimize  technology  and  system  development  program  investment 
strategies  in  order  to  ensure  budget  and  schedule  constraints  are  met.  Finally,  a  space  transportation 
system  sizing  environment  is  under  development  that  will  provide  a  technical  feasibility  assessment  to 
complement  the  viability  assessment  performed  by  the  architecture  evolution  modeling. 
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